Skip to content

ci: point uv at the baked venv so tests pass on the /next runner image - #509

Open
HarikrishnanBalagopal wants to merge 1 commit into
torch-spyre:mainfrom
HarikrishnanBalagopal:hfa-uv-venv-main
Open

ci: point uv at the baked venv so tests pass on the /next runner image#509
HarikrishnanBalagopal wants to merge 1 commit into
torch-spyre:mainfrom
HarikrishnanBalagopal:hfa-uv-venv-main

Conversation

@HarikrishnanBalagopal

@HarikrishnanBalagopal HarikrishnanBalagopal commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

What this does

The ARC self-hosted runner sets have been switched to boot from the
icr.io/ai_sw_accel/2.0/next/<product>:amd64-dev images (spyre-frameworks
merge, live). Those images ship the project venv baked and activated at
/home/senuser/.venv. With no hint to uv, uv run re-creates an empty project
.venv and re-syncs on every card job, so each Spyre test breaks at setup.

This sets uv to reuse the baked venv:

  • _test_matrix.yaml top-level env: gains UV_NO_SYNC: "1" and
    UV_PROJECT_ENVIRONMENT: /home/senuser/.venv, so uv run points at the baked
    venv and skips its implicit sync.
  • The check uv.lock job (ubuntu-latest, no baked venv) blanks both vars at
    job level, so the top-level values do not leak into setup-uv@v5 and fail it.
  • The two make model-module-tests steps drop the now-redundant
    source .venv/bin/activate.
  • gather-runner-info echoes VIRTUAL_ENV / UV_NO_SYNC /
    UV_PROJECT_ENVIRONMENT for diagnostics.

Why at the workflow level

This mirrors spyre-frameworks PR #1390, which bakes the same two vars into the
image layer. That PR is not yet merged, so the workflow-level fix keeps CI green
on the /next runners in the meantime. Once #1390 lands and the image carries
the vars, this top-level env: can be removed.

Issue

https://github.com/torch-spyre/hf-adapters/actions/runs/34507171144/job/102972671665?pr=183#step:5:22

ImportError while loading conftest '/home/senuser/hf-adapters/hf-adapters/tests/conftest.py'.
tests/conftest.py:55: in <module>
    from transformers import AutoModelForCausalLM, PretrainedConfig
E   ModuleNotFoundError: No module named 'transformers'
make: *** [Makefile:107: embed-compare-tests] Error 4

Validation

The identical change was validated on the shadow branch
(shadow-resource-validation-v2, PR #329) against the fresh /next image: the
card suites went from all-red-at-setup to green (~124 card jobs), the 3 VLM e2e
legs passed, and the only residual reds were image-independent (a Wikimedia HTTP
429 on the CLIP fixture download, which also flakes on the legacy control, and
the check uv.lock env leak this PR fixes). Shadow run:
https://github.com/torch-spyre/hf-adapters/actions/runs/34478885987

This is the non-shadow production version: it excludes the shadow-only
image_torch_spyre_shadow label swap in test_pull_request.yaml, so it targets
the real prod runner labels.

The ARC runner sets now boot from the icr.io/ai_sw_accel/2.0/next/<product>
images, which ship the project venv baked and activated at /home/senuser/.venv.
Without telling uv about it, `uv run` re-creates an empty project .venv and
re-syncs on every card job, so each Spyre test breaks at setup.

Set UV_NO_SYNC=1 and UV_PROJECT_ENVIRONMENT=/home/senuser/.venv at the
_test_matrix.yaml top level so uv reuses the baked venv and skips the implicit
sync. Neutralize both on the ubuntu-hosted check-uv-lock job, which has no baked
venv and would otherwise fail inside setup-uv. Drop the now-redundant
`source .venv/bin/activate` from the two model-module-tests steps. Echo
VIRTUAL_ENV / UV_NO_SYNC / UV_PROJECT_ENVIRONMENT from gather-runner-info for
diagnostics.

This mirrors spyre-frameworks PR #1390, which bakes the same vars into the image
layer; the workflow-level fix keeps CI green on the /next runners until that
image rebuild lands.

Signed-off-by: Harikrishnan Balagopal <Harikrishnan.Balagopal@ibm.com>

@BenjSz BenjSz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

⚙️ Auto-merge enabled — dispatched /spyre-test for 2dfac342 because the required Spyre Test gate had not run on this commit.

Automated: enabling auto-merge requests the gate, so the PR is not left waiting on a check nobody asked for. Re-enabling auto-merge on this same commit will not dispatch again.

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

⚠️ This run was superseded by a newer spyre-test run for this PR and has stood down.

❌ spyre-test: failure

Triggered by: /spyre-test comment

Plan (build waves + dependencies, per arch)

amd64

flowchart LR
  subgraph Lamd64_0["amd64 L0 · 1 parallel"]
    n_amd64_torch_spyre_torch_spyre_dev["torch-spyre/torch-spyre-dev 🔴<br/>image · 16ec4926d4c8"]
  end
  subgraph Lamd64_1["amd64 L1 · 1 parallel"]
    n_amd64_hf_adapters_hf_adapters_dev["hf-adapters/hf-adapters-dev 🟢<br/>image · 0aa666e8acc0"]
  end
  subgraph Lamd64_2["amd64 L2 · 1 parallel"]
    n_amd64_spyre_inference_spyre_inference_dev["spyre-inference/spyre-inference-dev 🔴<br/>image · b8b89d99752a"]
  end
  n_amd64_torch_spyre_torch_spyre_dev --> n_amd64_hf_adapters_hf_adapters_dev
  n_amd64_hf_adapters_hf_adapters_dev --> n_amd64_spyre_inference_spyre_inference_dev
  classDef sPending fill:#eceff1,stroke:#90a4ae,color:#37474f
  classDef sBuilding fill:#fff8e1,stroke:#f9a825,color:#5d4037,stroke-width:2px
  classDef sOk fill:#e8f5e9,stroke:#43a047,color:#1b5e20
  classDef sReused fill:#e3f2fd,stroke:#1e88e5,color:#0d47a1
  classDef sFailed fill:#ffebee,stroke:#e53935,color:#b71c1c,stroke-width:2px
  classDef sDropped fill:#f5f5f5,stroke:#bdbdbd,color:#9e9e9e
  class n_amd64_torch_spyre_torch_spyre_dev sOk;
  class n_amd64_hf_adapters_hf_adapters_dev sOk;
  class n_amd64_spyre_inference_spyre_inference_dev sOk;
Loading

✅ orch trigger-pr-validationgreen · arches amd64 · fp amd64=5d4c7455

level component arch build smoke unit integration trunk regression perf
L0 torch-spyre/torch-spyre-dev amd64 ✅ ok · 🟡 gha · · ·
L1 hf-adapters/hf-adapters-dev amd64 ✅ ok 🟢 · 🟢 gha · · ·
L2 spyre-inference/spyre-inference-dev amd64 ✅ ok 🟢 · 🟡 gha · · ·

GHA test runs:


⚠️ advisory failures only — mergeable, see below

Build: built 3

Tests: passed 3 · blocking 0 · advisory 2 · infra/inconclusive 0 · no signal 0

Failures by kind
  • ⚠️ advisory (does not block) · torch-spyre/amd64 integration: UNSTABLE
  • ⚠️ advisory (does not block) · torch-spyre/amd64 gha:integration: FAILURE
  • ℹ️ informational · spyre-inference/amd64 gha:integration: FAILURE

Before merging, consider:

  • torch-spyre/amd64 gha:integration, torch-spyre/amd64 integration failed with gating: "unstable" — advisory. It set the build UNSTABLE but does not block the merge. Worth a look, not a stop.
  • spyre-inference/amd64 gha:integration failed with gating: false — informational only, no merge impact.

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

GHA test runs:

@ashokponkumar

Copy link
Copy Markdown
Collaborator

/spyre-test

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

❌ spyre-test: failure

Triggered by: /spyre-test comment: #509 (comment)

Plan (build waves + dependencies, per arch)

amd64

flowchart LR
  subgraph Lamd64_0["amd64 L0 · 1 parallel"]
    n_amd64_spyre_backend_spyre_backend_dev["spyre-backend/spyre-backend-dev ↺<br/>image · eddfefe0e289"]
  end
  subgraph Lamd64_1["amd64 L1 · 1 parallel"]
    n_amd64_torch_spyre_torch_spyre_dev["torch-spyre/torch-spyre-dev 🔴 ↺<br/>image · 09ec21ef38de"]
  end
  subgraph Lamd64_2["amd64 L2 · 1 parallel"]
    n_amd64_hf_adapters_hf_adapters_dev["hf-adapters/hf-adapters-dev 🔴<br/>image · 00f2afeac7d0"]
  end
  subgraph Lamd64_3["amd64 L3 · 1 parallel"]
    n_amd64_spyre_inference_spyre_inference_dev["spyre-inference/spyre-inference-dev 🔴<br/>image · 94e8d8cd9da3"]
  end
  n_amd64_spyre_backend_spyre_backend_dev --> n_amd64_torch_spyre_torch_spyre_dev
  n_amd64_torch_spyre_torch_spyre_dev --> n_amd64_hf_adapters_hf_adapters_dev
  n_amd64_hf_adapters_hf_adapters_dev --> n_amd64_spyre_inference_spyre_inference_dev
  classDef sPending fill:#eceff1,stroke:#90a4ae,color:#37474f
  classDef sBuilding fill:#fff8e1,stroke:#f9a825,color:#5d4037,stroke-width:2px
  classDef sOk fill:#e8f5e9,stroke:#43a047,color:#1b5e20
  classDef sReused fill:#e3f2fd,stroke:#1e88e5,color:#0d47a1
  classDef sFailed fill:#ffebee,stroke:#e53935,color:#b71c1c,stroke-width:2px
  classDef sDropped fill:#f5f5f5,stroke:#bdbdbd,color:#9e9e9e
  class n_amd64_spyre_backend_spyre_backend_dev sReused;
  class n_amd64_torch_spyre_torch_spyre_dev sReused;
  class n_amd64_hf_adapters_hf_adapters_dev sOk;
  class n_amd64_spyre_inference_spyre_inference_dev sOk;
Loading

✅ orch trigger-pr-validationgreen · arches amd64 · fp amd64=059f8ea6

level component arch build smoke unit integration trunk regression perf
L0 spyre-backend/spyre-backend-dev amd64 ✅ ok (reused) · · · · · ·
L1 torch-spyre/torch-spyre-dev amd64 ✅ ok (reused) 🟢 · 🟡 gha · · ·
L2 hf-adapters/hf-adapters-dev amd64 ✅ ok 🟢 · 🟡 gha · · ·
L3 spyre-inference/spyre-inference-dev amd64 ✅ ok 🟢 · 🟡 gha · · ·

GHA test runs:


⚠️ advisory failures only — mergeable, see below

Build: built 2 · reused 2

Tests: passed 1 · blocking 0 · advisory 4 · infra/inconclusive 0 · no signal 0

Failures by kind
  • ⚠️ advisory (does not block) · torch-spyre/amd64 integration: UNSTABLE
  • ⚠️ advisory (does not block) · torch-spyre/amd64 gha:integration: FAILURE
  • ⚠️ advisory (does not block) · hf-adapters/amd64 integration: UNSTABLE
  • ⚠️ advisory (does not block) · hf-adapters/amd64 gha:integration: FAILURE
  • ℹ️ informational · spyre-inference/amd64 gha:integration: FAILURE

Before merging, consider:

  • hf-adapters/amd64 gha:integration, hf-adapters/amd64 integration, torch-spyre/amd64 gha:integration, torch-spyre/amd64 integration failed with gating: "unstable" — advisory. It set the build UNSTABLE but does not block the merge. Worth a look, not a stop.
  • spyre-inference/amd64 gha:integration failed with gating: false — informational only, no merge impact.

@HarikrishnanBalagopal

Copy link
Copy Markdown
Collaborator Author

This change is complete and does what it says. On the live /next :amd64-dev runner image, the baked, pre-activated venv at /home/senuser/.venv made uv run --active pick a venv that uv sync never populated, so every card test failed collection with ModuleNotFoundError: No module named 'transformers'. Pointing UV_PROJECT_ENVIRONMENT at the baked venv (and UV_NO_SYNC=1) makes the two converge on one complete venv.

Result on the latest run (34496557541): 129 pass, 8 fail (down from a full-matrix collection failure without this change). The 8 remaining failures are all external to this PR:

  • 4 VLM e2e jobs (granite-vision-4.1-4b, Ministral-3-3B, each plus its pod-level retry) fail in DDC compile, not in Python:
    dxp_standalone ... returned non-zero exit status 1, Program verification failed for core N node sdsc_11, Immediate value out of boundary ... LX_MVLOOPCNT on a sdsc_fused__scaled_dot_product_... kernel. This is a known DDC compiler boundary defect whose fix has already merged in the toolchain and is being rebuilt into the runner image. It clears once the /next image rebakes with the new toolchain build. Not related to the venv change.
  • 1 seq-classification job (distilbert sst-2) fails at setup with a transient git clone error: RPC failed; curl 92 HTTP/2 stream ... not closed cleanly, fatal: early EOF. Network flake; a re-run recovers it.
  • 3 fan-in gates (run spyre unit tests x2, Spyre test matrix result) only aggregate the above.

No remaining failure is caused by this change. The venv fix itself is proven by the 129 green card jobs that previously could not even collect. Once the /next image carries the deeptools #4691 RPM, a re-run should be fully green.

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

GHA test runs:

@ashokponkumar

Copy link
Copy Markdown
Collaborator

/spyre-test

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

❌ spyre-test: failure

Triggered by: /spyre-test comment: #509 (comment)

Plan (build waves + dependencies, per arch)

amd64

flowchart LR
  subgraph Lamd64_0["amd64 L0 · 1 parallel"]
    n_amd64_spyre_backend_spyre_backend_dev["spyre-backend/spyre-backend-dev ↺<br/>image · b6af73edcd2c"]
  end
  subgraph Lamd64_1["amd64 L1 · 1 parallel"]
    n_amd64_torch_spyre_torch_spyre_dev["torch-spyre/torch-spyre-dev 🔴 ↺<br/>image · fc1752087216"]
  end
  subgraph Lamd64_2["amd64 L2 · 1 parallel"]
    n_amd64_hf_adapters_hf_adapters_dev["hf-adapters/hf-adapters-dev 🟢<br/>image · 7dc77fe364ea"]
  end
  subgraph Lamd64_3["amd64 L3 · 1 parallel"]
    n_amd64_spyre_inference_spyre_inference_dev["spyre-inference/spyre-inference-dev 🔴<br/>image · c28982d013ad"]
  end
  n_amd64_spyre_backend_spyre_backend_dev --> n_amd64_torch_spyre_torch_spyre_dev
  n_amd64_torch_spyre_torch_spyre_dev --> n_amd64_hf_adapters_hf_adapters_dev
  n_amd64_hf_adapters_hf_adapters_dev --> n_amd64_spyre_inference_spyre_inference_dev
  classDef sPending fill:#eceff1,stroke:#90a4ae,color:#37474f
  classDef sBuilding fill:#fff8e1,stroke:#f9a825,color:#5d4037,stroke-width:2px
  classDef sOk fill:#e8f5e9,stroke:#43a047,color:#1b5e20
  classDef sReused fill:#e3f2fd,stroke:#1e88e5,color:#0d47a1
  classDef sFailed fill:#ffebee,stroke:#e53935,color:#b71c1c,stroke-width:2px
  classDef sDropped fill:#f5f5f5,stroke:#bdbdbd,color:#9e9e9e
  class n_amd64_spyre_backend_spyre_backend_dev sReused;
  class n_amd64_torch_spyre_torch_spyre_dev sReused;
  class n_amd64_hf_adapters_hf_adapters_dev sOk;
  class n_amd64_spyre_inference_spyre_inference_dev sOk;
Loading

✅ orch trigger-pr-validationgreen · arches amd64 · fp amd64=913da65a

level component arch build smoke unit integration trunk regression perf
L0 spyre-backend/spyre-backend-dev amd64 ✅ ok (reused) · · · · · ·
L1 torch-spyre/torch-spyre-dev amd64 ✅ ok (reused) 🟢 · 🟡 gha · · ·
L2 hf-adapters/hf-adapters-dev amd64 ✅ ok 🟢 · 🟢 gha · · ·
L3 spyre-inference/spyre-inference-dev amd64 ✅ ok 🟢 · 🟡 gha · · ·

GHA test runs:


⚠️ advisory failures only — mergeable, see below

Build: built 2 · reused 2

Tests: passed 3 · blocking 0 · advisory 2 · infra/inconclusive 0 · no signal 0

Failures by kind
  • ⚠️ advisory (does not block) · torch-spyre/amd64 integration: UNSTABLE
  • ⚠️ advisory (does not block) · torch-spyre/amd64 gha:integration: FAILURE
  • ℹ️ informational · spyre-inference/amd64 gha:integration: FAILURE

Before merging, consider:

  • torch-spyre/amd64 gha:integration, torch-spyre/amd64 integration failed with gating: "unstable" — advisory. It set the build UNSTABLE but does not block the merge. Worth a look, not a stop.
  • spyre-inference/amd64 gha:integration failed with gating: false — informational only, no merge impact.

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown

GHA test runs:

@ashokponkumar

Copy link
Copy Markdown
Collaborator

/spyre-test

@spyre-ci

spyre-ci Bot commented Sep 11, 2026

Copy link
Copy Markdown

❌ spyre-test: failure

Triggered by: /spyre-test comment: #509 (comment)

Plan (build waves + dependencies, per arch)

amd64

flowchart LR
  subgraph Lamd64_0["amd64 L0 · 1 parallel"]
    n_amd64_torch_spyre_torch_spyre_dev["torch-spyre/torch-spyre-dev 🔴 ↺<br/>image · 77b4e1902f25"]
  end
  subgraph Lamd64_1["amd64 L1 · 1 parallel"]
    n_amd64_hf_adapters_hf_adapters_dev["hf-adapters/hf-adapters-dev 🟢<br/>image · 4f42807b1666"]
  end
  subgraph Lamd64_2["amd64 L2 · 1 parallel"]
    n_amd64_spyre_inference_spyre_inference_dev["spyre-inference/spyre-inference-dev 🔴<br/>image · dfe6984e6f57"]
  end
  n_amd64_torch_spyre_torch_spyre_dev --> n_amd64_hf_adapters_hf_adapters_dev
  n_amd64_hf_adapters_hf_adapters_dev --> n_amd64_spyre_inference_spyre_inference_dev
  classDef sPending fill:#eceff1,stroke:#90a4ae,color:#37474f
  classDef sBuilding fill:#fff8e1,stroke:#f9a825,color:#5d4037,stroke-width:2px
  classDef sOk fill:#e8f5e9,stroke:#43a047,color:#1b5e20
  classDef sReused fill:#e3f2fd,stroke:#1e88e5,color:#0d47a1
  classDef sFailed fill:#ffebee,stroke:#e53935,color:#b71c1c,stroke-width:2px
  classDef sDropped fill:#f5f5f5,stroke:#bdbdbd,color:#9e9e9e
  class n_amd64_torch_spyre_torch_spyre_dev sReused;
  class n_amd64_hf_adapters_hf_adapters_dev sOk;
  class n_amd64_spyre_inference_spyre_inference_dev sOk;
Loading

✅ orch trigger-pr-validationgreen · arches amd64 · fp amd64=3e0495e1

level component arch build smoke unit integration trunk regression perf
L0 torch-spyre/torch-spyre-dev amd64 ✅ ok (reused) 🟢 · 🟡 gha · · ·
L1 hf-adapters/hf-adapters-dev amd64 ✅ ok 🟢 · 🟢 gha · · ·
L2 spyre-inference/spyre-inference-dev amd64 ✅ ok 🟢 · 🟡 gha · · ·

GHA test runs:


⚠️ advisory failures only — mergeable, see below

Build: built 2 · reused 1

Tests: passed 3 · blocking 0 · advisory 2 · infra/inconclusive 0 · no signal 0

Failures by kind
  • ⚠️ advisory (does not block) · torch-spyre/amd64 integration: UNSTABLE
  • ⚠️ advisory (does not block) · torch-spyre/amd64 gha:integration: FAILURE
  • ℹ️ informational · spyre-inference/amd64 gha:integration: FAILURE

Before merging, consider:

  • torch-spyre/amd64 gha:integration, torch-spyre/amd64 integration failed with gating: "unstable" — advisory. It set the build UNSTABLE but does not block the merge. Worth a look, not a stop.
  • spyre-inference/amd64 gha:integration failed with gating: false — informational only, no merge impact.

@spyre-ci

spyre-ci Bot commented Sep 11, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 11, 2026

Copy link
Copy Markdown

GHA test runs:

@spyre-ci

spyre-ci Bot commented Sep 11, 2026

Copy link
Copy Markdown

GHA test runs:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants